Labelling imaging datasets on the basis of neuroradiology reports: a validation study

Natural language processing (NLP) shows promise as a means to automate the labelling of hospital-scale neuroradiology magnetic resonance imaging (MRI) datasets for computer vision applications. To date, however, there has been no thorough investigation into the validity of this approach, including determining the accuracy of report labels compared to image labels as well as examining the performance of non-specialist labellers. In this work, we draw on the experience of a team of neuroradiologists who labelled over 5000 MRI neuroradiology reports as part of a project to build a dedicated deep learning-based neuroradiology report classifier. We show that, in our experience, assigning binary labels (i.e. normal vs abnormal) to images from reports alone is highly accurate. In contrast to the binary labels, however, the accuracy of more granular labelling is dependent on the category, and we highlight reasons for this discrepancy. We also show that downstream model performance is reduced when labelling of training reports is performed by a non-specialist. To allow other researchers to accelerate their research, we make our refined abnormality definitions and labelling rules available, as well as our easy-to-use radiology report labelling app which helps streamline this process

Effects of antiplatelet therapy on stroke risk by brain imaging features of intracerebral haemorrhage and cerebral small vessel diseases: subgroup analyses of the RESTART randomised, open-label trial

Background Findings from the RESTART trial suggest that starting antiplatelet therapy might reduce the risk of recurrent symptomatic intracerebral haemorrhage compared with avoiding antiplatelet therapy. Brain imaging features of intracerebral haemorrhage and cerebral small vessel diseases (such as cerebral microbleeds) are associated with greater risks of recurrent intracerebral haemorrhage. We did subgroup analyses of the RESTART trial to explore whether these brain imaging features modify the effects of antiplatelet therapy

Effects of antiplatelet therapy after stroke due to intracerebral haemorrhage (RESTART): a randomised, open-label trial

Background: Antiplatelet therapy reduces the risk of major vascular events for people with occlusive vascular disease, although it might increase the risk of intracranial haemorrhage. Patients surviving the commonest subtype of intracranial haemorrhage, intracerebral haemorrhage, are at risk of both haemorrhagic and occlusive vascular events, but whether antiplatelet therapy can be used safely is unclear. We aimed to estimate the relative and absolute effects of antiplatelet therapy on recurrent intracerebral haemorrhage and whether this risk might exceed any reduction of occlusive vascular events. Methods: The REstart or STop Antithrombotics Randomised Trial (RESTART) was a prospective, randomised, open-label, blinded endpoint, parallel-group trial at 122 hospitals in the UK. We recruited adults (≥18 years) who were taking antithrombotic (antiplatelet or anticoagulant) therapy for the prevention of occlusive vascular disease when they developed intracerebral haemorrhage, discontinued antithrombotic therapy, and survived for 24 h. Computerised randomisation incorporating minimisation allocated participants (1:1) to start or avoid antiplatelet therapy. We followed participants for the primary outcome (recurrent symptomatic intracerebral haemorrhage) for up to 5 years. We analysed data from all randomised participants using Cox proportional hazards regression, adjusted for minimisation covariates. This trial is registered with ISRCTN (number ISRCTN71907627). Findings: Between May 22, 2013, and May 31, 2018, 537 participants were recruited a median of 76 days (IQR 29–146) after intracerebral haemorrhage onset: 268 were assigned to start and 269 (one withdrew) to avoid antiplatelet therapy. Participants were followed for a median of 2·0 years (IQR [1·0– 3·0]; completeness 99·3%). 12 (4%) of 268 participants allocated to antiplatelet therapy had recurrence of intracerebral haemorrhage compared with 23 (9%) of 268 participants allocated to avoid antiplatelet therapy (adjusted hazard ratio 0·51 [95% CI 0·25–1·03]; p=0·060). 18 (7%) participants allocated to antiplatelet therapy experienced major haemorrhagic events compared with 25 (9%) participants allocated to avoid antiplatelet therapy (0·71 [0·39–1·30]; p=0·27), and 39 [15%] participants allocated to antiplatelet therapy had major occlusive vascular events compared with 38 [14%] allocated to avoid antiplatelet therapy (1·02 [0·65–1·60]; p=0·92). Interpretation: These results exclude all but a very modest increase in the risk of recurrent intracerebral haemorrhage with antiplatelet therapy for patients on antithrombotic therapy for the prevention of occlusive vascular disease when they developed intracerebral haemorrhage. The risk of recurrent intracerebral haemorrhage is probably too small to exceed the established benefits of antiplatelet therapy for secondary prevention

Effects of antiplatelet therapy after stroke due to intracerebral haemorrhage (RESTART): a randomised, open-label trial

Background: Antiplatelet therapy reduces the risk of major vascular events for people with occlusive vascular disease, although it might increase the risk of intracranial haemorrhage. Patients surviving the commonest subtype of intracranial haemorrhage, intracerebral haemorrhage, are at risk of both haemorrhagic and occlusive vascular events, but whether antiplatelet therapy can be used safely is unclear. We aimed to estimate the relative and absolute effects of antiplatelet therapy on recurrent intracerebral haemorrhage and whether this risk might exceed any reduction of occlusive vascular events. Methods: The REstart or STop Antithrombotics Randomised Trial (RESTART) was a prospective, randomised, open-label, blinded endpoint, parallel-group trial at 122 hospitals in the UK. We recruited adults (≥18 years) who were taking antithrombotic (antiplatelet or anticoagulant) therapy for the prevention of occlusive vascular disease when they developed intracerebral haemorrhage, discontinued antithrombotic therapy, and survived for 24 h. Computerised randomisation incorporating minimisation allocated participants (1:1) to start or avoid antiplatelet therapy. We followed participants for the primary outcome (recurrent symptomatic intracerebral haemorrhage) for up to 5 years. We analysed data from all randomised participants using Cox proportional hazards regression, adjusted for minimisation covariates. This trial is registered with ISRCTN (number ISRCTN71907627). Findings: Between May 22, 2013, and May 31, 2018, 537 participants were recruited a median of 76 days (IQR 29–146) after intracerebral haemorrhage onset: 268 were assigned to start and 269 (one withdrew) to avoid antiplatelet therapy. Participants were followed for a median of 2·0 years (IQR [1·0– 3·0]; completeness 99·3%). 12 (4%) of 268 participants allocated to antiplatelet therapy had recurrence of intracerebral haemorrhage compared with 23 (9%) of 268 participants allocated to avoid antiplatelet therapy (adjusted hazard ratio 0·51 [95% CI 0·25–1·03]; p=0·060). 18 (7%) participants allocated to antiplatelet therapy experienced major haemorrhagic events compared with 25 (9%) participants allocated to avoid antiplatelet therapy (0·71 [0·39–1·30]; p=0·27), and 39 [15%] participants allocated to antiplatelet therapy had major occlusive vascular events compared with 38 [14%] allocated to avoid antiplatelet therapy (1·02 [0·65–1·60]; p=0·92). Interpretation: These results exclude all but a very modest increase in the risk of recurrent intracerebral haemorrhage with antiplatelet therapy for patients on antithrombotic therapy for the prevention of occlusive vascular disease when they developed intracerebral haemorrhage. The risk of recurrent intracerebral haemorrhage is probably too small to exceed the established benefits of antiplatelet therapy for secondary prevention

Effects of antiplatelet therapy after stroke due to intracerebral haemorrhage (RESTART): a randomised, open-label trial

BACKGROUND: Antiplatelet therapy reduces the risk of major vascular events for people with occlusive vascular disease, although it might increase the risk of intracranial haemorrhage. Patients surviving the commonest subtype of intracranial haemorrhage, intracerebral haemorrhage, are at risk of both haemorrhagic and occlusive vascular events, but whether antiplatelet therapy can be used safely is unclear. We aimed to estimate the relative and absolute effects of antiplatelet therapy on recurrent intracerebral haemorrhage and whether this risk might exceed any reduction of occlusive vascular events. METHODS: The REstart or STop Antithrombotics Randomised Trial (RESTART) was a prospective, randomised, open-label, blinded endpoint, parallel-group trial at 122 hospitals in the UK. We recruited adults (≥18 years) who were taking antithrombotic (antiplatelet or anticoagulant) therapy for the prevention of occlusive vascular disease when they developed intracerebral haemorrhage, discontinued antithrombotic therapy, and survived for 24 h. Computerised randomisation incorporating minimisation allocated participants (1:1) to start or avoid antiplatelet therapy. We followed participants for the primary outcome (recurrent symptomatic intracerebral haemorrhage) for up to 5 years. We analysed data from all randomised participants using Cox proportional hazards regression, adjusted for minimisation covariates. This trial is registered with ISRCTN (number ISRCTN71907627). FINDINGS: Between May 22, 2013, and May 31, 2018, 537 participants were recruited a median of 76 days (IQR 29-146) after intracerebral haemorrhage onset: 268 were assigned to start and 269 (one withdrew) to avoid antiplatelet therapy. Participants were followed for a median of 2·0 years (IQR [1·0- 3·0]; completeness 99·3%). 12 (4%) of 268 participants allocated to antiplatelet therapy had recurrence of intracerebral haemorrhage compared with 23 (9%) of 268 participants allocated to avoid antiplatelet therapy (adjusted hazard ratio 0·51 [95% CI 0·25-1·03]; p=0·060). 18 (7%) participants allocated to antiplatelet therapy experienced major haemorrhagic events compared with 25 (9%) participants allocated to avoid antiplatelet therapy (0·71 [0·39-1·30]; p=0·27), and 39 [15%] participants allocated to antiplatelet therapy had major occlusive vascular events compared with 38 [14%] allocated to avoid antiplatelet therapy (1·02 [0·65-1·60]; p=0·92). INTERPRETATION: These results exclude all but a very modest increase in the risk of recurrent intracerebral haemorrhage with antiplatelet therapy for patients on antithrombotic therapy for the prevention of occlusive vascular disease when they developed intracerebral haemorrhage. The risk of recurrent intracerebral haemorrhage is probably too small to exceed the established benefits of antiplatelet therapy for secondary prevention. FUNDING: British Heart Foundation

Acute ischaemic brain lesions in intracerebral haemorrhage : multicentre cross-sectional magnetic resonance imaging study.

Subclinical acute ischaemic lesions on brain magnetic resonance imaging have recently been described in spontaneous intracerebral haemorrhage, and may be important to understand pathophysiology and guide treatment. The underlying mechanisms are uncertain. We tested the hypothesis that ischaemic lesions are related to magnetic resonance imaging markers of the severity and type of small-vessel disease (hypertensive arteriopathy or cerebral amyloid angiopathy) in a multicentre, cross-sectional study. We studied consecutive patients with intracerebral haemorrhage from four specialist stroke centres, and age-matched stroke service referrals without intracerebral haemorrhage. Acute ischaemic lesions were assessed on magnetic resonance imaging (<3 months after intracerebral haemorrhage) using diffusion-weighted imaging. White matter changes and cerebral microbleeds were rated with validated scales. We investigated associations between diffusion-weighted imaging lesions, clinical and radiological characteristics. We included 114 patients with intracerebral haemorrhage (39 with clinically probable cerebral amyloid angiopathy) and 47 age-matched controls. The prevalence of diffusion-weighted imaging lesions was 9/39 (23%) in probable cerebral amyloid angiopathy-related intracerebral haemorrhage versus 6/75 (8%) in the remaining patients with intracerebral haemorrhage (P = 0.024); no diffusion-weighted imaging lesions were found in controls. Diffusion-weighted imaging lesions were mainly cortical and were associated with mean white matter change score (odds ratio 1.14 per unit increase, 95% confidence interval 1.02-1.28, P = 0.024) and the presence of strictly lobar cerebral microbleeds (odds ratio 3.85, 95% confidence interval 1.15-12.93, P = 0.029). Acute, subclinical ischaemic brain lesions are frequent but previously underestimated after intracerebral haemorrhage, and are three times more common in cerebral amyloid angiopathy-related intracerebral haemorrhage than in other intracerebral haemorrhage types. Ischaemic brain lesions are associated with white matter changes and cerebral microbleeds, suggesting that they result from an occlusive small-vessel arteriopathy. Diffusion-weighted imaging lesions contribute to the overall burden of vascular-related brain damage in intracerebral haemorrhage, and may be a useful surrogate marker of ongoing ischaemic injury from small-vessel damage

Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection

Unlocking the vast potential of deep learning-based computer vision classification systems necessitates large data sets for model training. Natural Language Processing (NLP)—involving automation of dataset labelling—represents a potential avenue to achieve this. However, many aspects of NLP for dataset labelling remain unvalidated. Expert radiologists manually labelled over 5,000 MRI head reports in order to develop a deep learning-based neuroradiology NLP report classifier. Our results demonstrate that binary labels (normal vs. abnormal) showed high rates of accuracy, even when only two MRI sequences (T2-weighted and those based on diffusion weighted imaging) were employed as opposed to all sequences in an examination. Meanwhile, the accuracy of more specific labelling for multiple disease categories was variable and dependent on the category. Finally, resultant model performance was shown to be dependent on the expertise of the original labeller, with worse performance seen with non-expert vs. expert labellers

Automated Labelling using an Attention model for Radiology reports of MRI scans (ALARM)

Labelling large datasets for training high-capacity neural networks is a major obstacle to the development of deep learning-based medical imaging applications. Here we present a transformer-based network for magnetic resonance imaging (MRI) radiology report classification which automates this task by assigning image labels on the basis of free-text expert radiology reports. Our model's performance is comparable to that of an expert radiologist, and better than that of an expert physician, demonstrating the feasibility of this approach. We make code available online for researchers to label their own MRI datasets for medical imaging applications

Deep learning to automate the labelling of head MRI datasets for computer vision applications

OBJECTIVES: The purpose of this study was to build a deep learning model to derive labels from neuroradiology reports and assign these to the corresponding examinations, overcoming a bottleneck to computer vision model development. METHODS: Reference-standard labels were generated by a team of neuroradiologists for model training and evaluation. Three thousand examinations were labelled for the presence or absence of any abnormality by manually scrutinising the corresponding radiology reports (‘reference-standard report labels’); a subset of these examinations (n = 250) were assigned ‘reference-standard image labels’ by interrogating the actual images. Separately, 2000 reports were labelled for the presence or absence of 7 specialised categories of abnormality (acute stroke, mass, atrophy, vascular abnormality, small vessel disease, white matter inflammation, encephalomalacia), with a subset of these examinations (n = 700) also assigned reference-standard image labels. A deep learning model was trained using labelled reports and validated in two ways: comparing predicted labels to (i) reference-standard report labels and (ii) reference-standard image labels. The area under the receiver operating characteristic curve (AUC-ROC) was used to quantify model performance. Accuracy, sensitivity, specificity, and F1 score were also calculated. RESULTS: Accurate classification (AUC-ROC > 0.95) was achieved for all categories when tested against reference-standard report labels. A drop in performance (ΔAUC-ROC > 0.02) was seen for three categories (atrophy, encephalomalacia, vascular) when tested against reference-standard image labels, highlighting discrepancies in the original reports. Once trained, the model assigned labels to 121,556 examinations in under 30 min. CONCLUSIONS: Our model accurately classifies head MRI examinations, enabling automated dataset labelling for downstream computer vision applications. KEY POINTS: • Deep learning is poised to revolutionise image recognition tasks in radiology; however, a barrier to clinical adoption is the difficulty of obtaining large labelled datasets for model training. • We demonstrate a deep learning model which can derive labels from neuroradiology reports and assign these to the corresponding examinations at scale, facilitating the development of downstream computer vision models. • We rigorously tested our model by comparing labels predicted on the basis of neuroradiology reports with two sets of reference-standard labels: (1) labels derived by manually scrutinising each radiology report and (2) labels derived by interrogating the actual images. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s00330-021-08132-0